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Abstract. The properties of incommensurate antiferromagnetic (AFM) order in the 
Fulde-Ferrell-Larkin-Ovchinnikov (FFLO) state is studied by solving the Bogoliubov- 
de-Gennes (BdG) equations. The relationship between the electronic structure and 
the magnetic structure is clarified. We find that the magnetic structure in the AFM- 
FFLO state includes three cases. (I) In the strongly localized case, the AFM staggered 
moment is confined into the FFLO nodal planes where the superconducting order 
parameter vanishes. (II) In the weakly localized case, the AFM staggered moment 
appears in the whole spatial region, and its magnitude is enhanced around the FFLO 
nodal planes. (Ill) In the extended case, the AFM staggered moment is nearly 
homogeneous and slightly suppressed in the vicinity of FFLO nodal planes. The 
structure of Bragg peaks in the momentum resolved structure factor is studied in 
each case. We discuss the possibility of AFM-FFLO state in the heavy fermion 
superconductor CeCoIns by comparing these results with the neutron scattering data of 
CeCoIns. Experimentally the magnetic structure and its dependence on the magnetic 
field orientation in the high field superconducting phase of CeCoIns are consistent with 
the case (II). 



1. Introduction 

The possible presence of a spatially modulated state in superconductors in a high 
magnetic field was predicted by Fulde and Ferrell [T] , and by Larkin and Ovchinnikov [2] 
more than 40 years ago. While the Bardeen-Cooper-Schrieffer (BCS) theory assume 
Cooper pairs with vanishing total momentum, the FFLO superconducting state 
represents a condensate of Cooper pairs with a finite total momentum. Since the 
FFLO state has an internal degree of freedom arising from the reflection or inversion 
symmetry, a spontaneous breaking of the spatial symmetry occurs. Although this novel 
superconducting state with an exotic symmetry has been attracting much interest, the 
FFLO state has not been observed in superconductors for nearly 40 years. Under 
these circumstances, the discovery of a new superconducting phase in CeCoIn 5 at high 
magnetic fields and low temperatures j3l 0] triggered many theoretical and experimental 
studies because this high-field superconducting (HFSC) phase is a likely candidate for 
the FFLO state [5]. The recent interest on the FFLO superconductivity/superfluidity 
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extends further into various related fields, such as organic superconductors [61 171 151 191 [TO] . 
cold atom gases [HI 02], astrophysics, and nuclear physics [13]. 

The HFSC phase of CeCoIn 5 has been interpreted widely within the concept of 
the FFLO state [51 HH [151 HSl [IZl HH1 HH [201 EU [221 [23]. However, recent observations 
of the magnetic order in the HFSC phase call for a reexamination of this conclusion 
[2l[ 125] . It is not unlikely that this order will be closely connected with a AFM quantum 
critical point observed in CeCoIn 5 [213127]. Moreover, the nuclear magnetic resonance 
(NMR) [2H EU [29] and neutron scattering [251 130] measurements may have uncovered 
a novel superconducting state in this strongly correlated electron system. 

Neutron scattering measurements have found that the wave vector of the AFM 
order is incommensurate <jaf = Q + Qmc with Q = (ir, n) and the AFM staggered 
moment Maf is oriented along the c-axis [23] ■ Recent experiments have shown that 
the incommensurability q[ nc is fixed along [1,-1,0] irrespective whether magnetic field is 
directed along [1,1,0] and [1,0,0] in the tetragonal lattice [30]. These magnetic structures 
are consistent with the NMR measurements [31] . 

Some theoretical scenarios have been proposed to explain the AFM order in the 
HFSC phase of CeCoIn 5 . We have analyzed the possibility that AFM order arises from 
the inhomogeneous Larkin-Ovchinnikov (LO) state [32| 133]. The AFM order triggered 
by the emergence of 7r-triplet pairing or pair density wave (PDW) has been investigated 
in the BCS state [MJ [351 EE] and in the homogeneous Fulde-Ferrell (FF) state [37]. In 
order to identify the HFSC phase of CeCoIn 5 it is highly desirable to examine these 
possible phases by comparing their properties with the experimental results. In this 
study, we investigate the magnetic structure of AFM-FFLO state, in which the AFM 
order appears in the inhomogeneous LO state, and discuss the recent neutron scattering 
measurements. 

2. Formulation 

Our theoretical analysis is based on the microscopic model, 

H = -t y ct c 7 +f y d c? +f y d c? -nYd c 7 

(?J},°- <<<* 3))),°- »V 

+ Uyn, t n u + Vy n.nj + jy § x §j - (1) 

* (*.J> (iJ) * 

where Sj is the spin operator and is the number operator at site % = (m,n). To 
describe the quasi-two-dimensional electronic structure of CeCoIn 5 , we assume a square 
lattice, in which the bracket (i, j), ((i , j )), and (((i , j ))) denote the summation over 
the nearest-neighbor sites, next-nearest-neighbor sites, and third nearest neighbor sites, 
respectively. 

The on-site repulsive interaction is given by U, and V and J stand for the 
attractive and AFM exchange interactions, respectively, between nearest-neighbor sites. 
We introduce V to stabilize the (i-wave superconducting state within the mean field 
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BdG equations and choose J > favoring AFM correlation in CeCoIns. The effective 
interaction leading to the <i-wave superconductivity near the AFM instability arises from 
the simple Hubbard model or periodical Anderson model beyond the mean field theory 
[38| [39] , but we here assume the interactions V and J to describe these features in the 
inhomogeneous LO phase on the basis of mean field BdG equations. Although the BdG 
equations neglect the AFM spin fluctuation beyond the mean field approximation, they 
are suitable for studying the qualitative features of the inhomogeneous superconducting 
and/or magnetic state. The roles of AFM spin fluctuation has been discussed in Ref. [33J. 

With the last term in eq. (1), we include the Zeeman coupling due to the applied 
magnetic field. The ^-factor is assumed to be (?b — 2. The magnetic field lies in the 
afr-plane of the tetragonal lattice and the superconducting vortices are neglected for 
simplicity. We choose the unit of energy such that t — 1 and t'/t = 0.25, t"/t = 0.05. 
The chemical potential enters as \i = fio + {\U + 4V)n Q , where n is the number density 
for U = V = J = H = 0. We vary the bare chemical potential from /i = —1.25 to 
/i = —1.15 to investigate the possible magnetic structures in the AFM-FFLO state. 
Then, we obtain the number density n = 0.72 ~ 0.82. 

We study the magnetic structure in the AFM-FFLO state in the following way. 
First, the BdG equations are self-consistently solved for the mean fields of the spin 
(S^), charge {n-?), and superconductivity A?^ = (c^cj^,), where is the spin operator 
parallel to the magnetic field. We take into account the Hartree term arising from U, V, 
and J. Since the spin singlet d-wecve state is stable in this model, the superconducting 
order parameter is described as 

A d (i)=AV: , + „-At^ --AlK r , (2) 

v ' i,i+a 1,1 — a i ,i +b b v ' 

with a and b being the unit vector along the a- and 6-axes, respectively. As for the 
spatial dependence of A d (i ), we find that the inhomogeneous LO state is stable against 
the uniform FF state. Then, the order parameter is real and oscillates in the space. 
We assume the single-g spatial modulation along the magnetic field in which the order 
parameter is approximately described as A d (i) ~ Acos(ig*FFLO • with g*FFLO || H 
except for the vicinity of the BCS-FFLO transition. This is the most stable FFLO state 
in the presence of superconducting vortices when the Fermi surface is nearly isotropic. 
We determine the stable superconducting state by comparing the condensation energy of 
BCS, FFLO and normal states. The orientation of magnetic field is taken into account 
in the direction of g*FFLO in our formulation. The case of g*FFLO not parallel to H will be 
discussed at the last of this paper. 

Second, we determine the magnetic instability by calculating the transverse spin 
susceptibility using the random phase approximation, 

XHI 3) = ^i i, J\ - - (3) 

l-£ S /(i, J') 

where I(i , i) = U, I(i , i ± a) = I(i , i ±b) = — ~, and otherwise I(i ,k) = 0. The bare 
spin susceptibility Xo(^i) i s calculated for the mean field Hamiltonian obtained by 
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solving the BdG equations. The magnetic instability is determined by the divergence 
of transverse spin susceptibility assuming a second order magnetic phase transition. 
Then, the criterion is A max = 1 where A max is the maximum eigenvalue of the matrix 
K(i,j) = J2k I(i > ^)Xo(^) J )• The magnetic moment perpendicular to the magnetic 
field is obtained as 

M(T) = I -1 (?,A;)m(£), (4) 

near the critical point, where m{k) is the eigenvector of K(i , j ) for the eigenvalue A max 
and X^/ _1 (i ,k)I(k, j) = We denote the AFM staggered moment M\ F (i) = 

(-l) m+n M(i) with % = (m,n). 

Since the effect of induced PDW order, which plays an important role to stabilize 
the AFM-FFLO state, is ignored in this formulation, the stability of the AFM order 
is underestimated. However, we confirmed that the induced PDW does not affect the 
magnetic structure [33] . Choosing the parameters U, V, and J so that A max = 1 in 
the FFLO state as in Refs. [321 E3], we discuss the variety of magnetic structures in 
the AFM-FFLO state in the following sections. In our model the incommensurate wave 
vector q[ nc is along a or b direction and different from the experimental observation 
q inc || [1,±1,0] [23]. However, the relationship between the magnetic structure and 
relative angle of q[ nc and g*FFLO (and H) is appropriately captured in our calculation. 

3. Magnetic structure 

We here show that the magnetic structure in the AFM-FFLO state is classified into three 
cases. (I) In the strongly localized case, the AFM moment is confined into the FFLO 
nodal planes where the superconducting order parameter vanishes. The magnitude 
of AFM moment away from the nodal planes is typically ~ 0.01 of that around the 
nodal planes. (II) In the weakly localized case, the AFM moment appears in the whole 
spatial region, and its magnitude is enhanced around the FFLO nodal planes, typically 
twice. (Ill) In the extended case, the AFM moment is nearly homogeneous and slightly 
suppressed in the vicinity of FFLO nodal planes. 

Owing to the four fold rotation symmetry of the tetragonal lattice the 
incommensurate wave vector q inc = q£ c || a is degenerate with q inc = q^ c \\ b in the 
normal state and uniform BCS state. This degeneracy is lifted in the FFLO state except 
for ^fflo || <finc + <?in C - Figures 1(a), 1(b), and 1(c) show typical magnetic structures 
of cases (I), (II), and (III), respectively for the magnetic field H \\ q£ c . We see that 
the modulation vector of AFM moment is different between the case (III) and the 
other two cases. In the extended case the incommensurate wave vector q[ nc is parallel 
to the FFLO modulation vector g*FFLO- Thus, the extended case (III) is incompatible 
with the neutron scattering measurement [25], assuming the relation H || ^fflo- On 
the other hand, the incommensurate wave vector q inc is perpendicular to the FFLO 
modulation vector ^fflo m the cases (I) and (II), consistent with the neutron scattering 
measurement for H || [1, ±1, 0]. This means that the neutron scattering measurement 
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Figure 1. The AFM staggered moment normalized by its maximum value 
Maf(0 /M max with i = (m,ri) in the case of <?fflo II Qinc- This FFLO modulation 
corresponds to the experimental setup of CeCoIns for the magnetic field along [1, ±1, 0] 
direction [25]. (a), (b), and (c) show (I) strongly localized case, (II) weakly localized 
case, and (III) extended case, respectively. We solve the BdG equations for 40 lattices. 
We choose the parameters (a) ^ = —1-15, U = 0.9, J = 0.6, and V = —0.5, (b) 
Ho = -1-19, U = 1.05, J = 0.6, and V = -0.5, and (c) fi = -1-25, U = 1.15, 
J = 0.65, and V = —0.5 so that the AFM order occurs in the FFLO state at (a) 
(T, H) = (0.0315,0.164), (b) (T,H) = (0.025,0.155), and (c) (T,H) = (0.02,0.168), 
respectively. The arrow shows the direction of magnetic field. 



indicates the localized AFM moment around the FFLO nodal planes. 

Figure 2 shows the magnetic structure for the magnetic field H \\ q^ c + q£ c . 
We see that the AFM staggered moment M\p(i) is distributed similarly to Fig. 1. 
The magnetic moment is strongly (weakly) localized in Fig. 2(a) (Fig. 2(b)), while 
that is extended in Fig. 2(c). Figs. 2(b) and 2(c) show double-g structure like 
Maf(0 ~ cos (<?inc ' i cos (ftnc * m contrast to Figs. 1(b) and 1(c). This double- 
q structure arises from the translational symmetry breaking in the FFLO state with 
9fflo || Vine + ftnc- We will discuss this point at the end of this paper. 

4. Neutron scattering 

For a comparison with neutron scattering experiments [25/30], we calculate the magnetic 
structure factor \M(q)\ 2 , where M(q) is the Fourier transformed magnetic moment given 

as 

M{q) = ^M(?)e if? . (5) 

i 

We here normalize M(i) so that Yjq\M(q)\ 2 = 1 to discuss the relative intensity of 
Bragg peaks. Figures 3 and 4 show the magnetic structure factor in the AFM-FFLO 
states with gpTLO II <?inc an d <?fflo II <?!nc + ftno respectively. The former corresponds to 
CeCoIn 5 under the magnetic field along [1, ±1, 0] direction, while the latter is realized in 
the magnetic field along [1, 0, 0] or [0, 1, 0] direction. We have shown the figures rotated 
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Figure 2. The normalized AFM staggered moment Mxp(i)/M max in the case of 
<Zfflo II Qinc + 9inc- This FFLO modulation corresponds to the CeCoIn 5 in the 
magnetic field along [1,0,0] or [0,1,0] direction [53J [30]. We assume the same 
parameter as in Fig. 1. The AFM order occurs at (a) (T, H) = (0.0264,0.164), (b) 
(T, H) = (0.0324,0.155), and (c) (T, H) = (0.035,0.15), respectively. We solve the 
BdG equations for 56 x 56 lattices to keep the amplitude of FFLO modulation vector 
|<Tfflo| similar to in Fig. 1. The arrow shows the direction of magnetic field. 



45 degree for a comparison with neutron scattering experiments [251 EQ] • 

In addition to the main Bragg peaks at q = q£ c , and/or q = q£ c , satellite peaks 
appear along the direction of magnetic field from the main peaks. These satellite peaks 
arise from the broken translational symmetry in the FFLO state, and their amplitude 
reflects the spatial inhomogeneity of magnetic structure. Therefore, the satellite peaks 
are pronounced in (I) the strongly localized case, while those are obscure in the cases 
(II) and (III). 

We here discuss the possible magnetic structure of CeCoIn 5 in the HFSC phase 
on the basis of Figs. 3 and 4. The strongly localized case (I) is incompatible with 
two features of neutron scattering experiments. First, the satellite peaks are absent 
or have a very weak intensity [23 I3"U] . Second, the position of main Bragg peaks is 
independent of the orientation of magnetic field in the afr-plane [30]. The extended 
case (III) is incompatible with the neutron scattering measurement for magnetic fields 
along [1, ±1, 0] for which the incommensurate wave vector g; nc is perpendicular to the 
field H [25J. On the other hand, the weakly localized case (II) shown in Figs. 3(b) and 
4(b) is consistent with the neutron scattering experiments in which the incommensurate 
wave vector is perpendicular to the field direction q inc _L H for H || [1,±1,0], and the 
position of main Bragg peaks does not change by rotating the magnetic field in the 
afr-plane. The four main Bragg peaks appear in Fig. 4(b) in contrast to Fig. 3(b) owing 
to the symmetry of system. This change has been observed in the neutron scattering 
measurement too [3D]- According to these discussions, the magnetic structure in the 




Figure 3. The magnetic structure factor \M(q)\ 2 in the AFM-FFLO state with 
<Zfflo || <?inc- This AFM-FFLO state can be realized in CeCoIn 5 for the magnetic 
field along [1,±1,0] direction [25]. We show the figures rotated by 45 degrees for 
a comparison with neutron scattering experiments. The center of each figure shows 
q = Q = (it, it), and therefore, the position from the center shows the incommensurate 
wave vector q[ nc . We assume the same parameters as in Fig. 1. 




Figure 4. The magnetic structure factor \M(q)\ 2 in the AFM-FFLO state with 
<Zfflo II <finc + 9inc- This AFM-FFLO state can be realized in CeCoIn 5 for the magnetic 
field along [1, 0, 0] or [0, 1, 0] direction [3D]. We assume the same parameters as in Fig. 3. 



possible AFM-FFLO state in CeCoIn 5 should be (II) the weakly localized case. This is 
the main conclusion of this paper. 



s 



5. Discussion 

We studied the magnetic structure of AFM order in the FFLO superconducting state. 
We find that the spatial inhomogeneity of magnetic moment is reduced by increasing the 
incommensurability |<£ nc | and enhancing the nesting of Fermi surface. This change of the 
magnetic structure is realized in our model by decreasing the number density from the 
half filling. Comparing our results with the neutron scattering experiments in CeCoIn 5 , 
the unconventional magnetic order in the HFSC phase of CeCoIns is consistent with the 
AFM-FFLO state proposed by us when the AFM staggered moment is weakly localized 
around the FFLO nodal planes. This is the "weakly localized case" in our classification 
of the magnetic structure. 

Finally, we discuss two points. (1) In our discussion we assumed that the FFLO 
modulation vector g*FFLO lies parallel to the magnetic field, since this is the most stable 
FFLO state, unless the anisotropy of Fermi surface favors another g*FFLO On the 
other hand, if ^fflo is fixed by the anisotropic electronic structure, the position of the 
Bragg peaks measured by neutron scattering would be independent of the magnetic 
field orientation, consistent with the experimental observation. This would apply 
to the AFM-FFLO state in the "weakly localized CclSG clS well as in the "strongly 
localized case" and the "extended case". This implies, however, that BCS-to-AFM- 
FFLO transition would be of first order |2U E2J [23] . While this is in contrast to the 
experimental observation, we can not exclude a weakly first order transition which is 
experimentally hard to detect. 

(2) The magnetic structure for the magnetic field along q£ c + q£ c corresponds 
to H || [1,0,0] in CeCoIn 5 . For this field direction Fig. 2 shows a double-g Bragg 
peak structure of the magnetization Maf(0 ~ cosine • i cos (<finc ' which is 
different from the single-g structure seen in Fig. 1. We distinguish two situations 
for the evolution of Maf(0 = rjacos^q^ ■ i) + r^cos^^ ■ i) ioi H \\ q£ c + q^ c . 
In the "commensurate case" where q£ c + q^ c = 2A^gpFLO, N being an integer, the 
double-g structure (\rj a \ = \r] b \) remains stable for a small temperature range below T N 
and then changes continuously towards a single-g structure, Maf(0 ~ cosine ' or 
Maf(0 ~ cosine ' 0; by shifting the weight between the two components, \rj a \ ^ \r) b \. 
Note that this corresponds to a symmetry breaking transition of Zi character. On the 
other hand, for the "incommensurate case" q£ c + q^ c ^ 2iV<fFFLO, we find i] a = or 
r] b = (single-g structure) for any T < T N . Since g*FFLO depends on the magnetic 
field, we expect to see a sequence of commensurate points following the phase boundary 
Tn(H) in the H-T phase diagram. This leads to a intriguing phase diagram near 
the commensurate points, which will be shown in another publication [30]. In most 
of the regime inside the AFM-FFLO phase the single-g state is stable for magnetic 
fields H || [1,0,0]. If this assumption holds true then the observed four Bragg peaks 
in neutron scattering should be interpreted as arising from domain formation of the 
two degenerate single-g states, i.e. \rj a \ > \r] b \ and |?7 a | < \rjb\- Note that the NMR 
spectrum in the HFSC phase of CeCoIn 5 [31] is consistent with a single-g structure, but 
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not with the double-g structure. The detailed analysis of the NMR experiments under 
the assumption of weakly localized form of the AFM-FFLO state, as proposed in this 
paper, will be reported elsewhere. 
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